{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cMwQC5WrEV2N"
   },
   "source": [
    "In this section we will be building more complex pipelines using mixed type columnar data. On the way, we will build custom encoders in sci-kit learn and a pipeline comprising all features and their preprocessing steps.  \n",
    "\n",
    "The Speed dating dataset accompanied the book *Data Analysis Using Regression and Multilevel/Hierarchical Models* by Andrew Gelman and Jennifer Hill. They used the examples to show applications for regressions within a Bayesian framework. The main technique covered was Bayesian inference using Gibbs Sampling, a technique that is now considered to be largely obsolete. We will use a different technique here, and we will have fun explaining some mechanics of human mate choices. You can find out more information [about this dataset at OpenML](https://www.openml.org/d/40536).\n",
    "\n",
    "\n",
    "The dataset is available at openML."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# making sure imbalanced-learn and scikit-learn are in compatible versions:\n",
    "!pip install -U 'scikit-learn>=0.23' 'imbalanced-learn==0.7.0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "tlmeJ8InwxZE",
    "outputId": "10e26bdb-3375-4f70-d5ea-273d8ac8d6f5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "!pip install -q openml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ahMOV0Omxq3k"
   },
   "outputs": [],
   "source": [
    "import openml\n",
    "dataset = openml.datasets.get_dataset(40536)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ok9QBxB_yIW2"
   },
   "outputs": [],
   "source": [
    "X, y, categorical_indicator, _ = dataset.get_data(\n",
    "    dataset_format='dataframe',\n",
    "    target=dataset.default_target_attribute\n",
    ")\n",
    "categorical_features = X.columns[categorical_indicator]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 423
    },
    "colab_type": "code",
    "id": "XlHVgTySD9P1",
    "outputId": "3a301550-14ed-417f-8424-b452e083ae02"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>has_null</th>\n",
       "      <th>wave</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>age_o</th>\n",
       "      <th>d_age</th>\n",
       "      <th>d_d_age</th>\n",
       "      <th>race</th>\n",
       "      <th>race_o</th>\n",
       "      <th>samerace</th>\n",
       "      <th>...</th>\n",
       "      <th>expected_num_interested_in_me</th>\n",
       "      <th>expected_num_matches</th>\n",
       "      <th>d_expected_happy_with_sd_people</th>\n",
       "      <th>d_expected_num_interested_in_me</th>\n",
       "      <th>d_expected_num_matches</th>\n",
       "      <th>like</th>\n",
       "      <th>guess_prob_liked</th>\n",
       "      <th>d_like</th>\n",
       "      <th>d_guess_prob_liked</th>\n",
       "      <th>met</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>female</td>\n",
       "      <td>21.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>[4-6]</td>\n",
       "      <td>Asian/Pacific Islander/Asian-American</td>\n",
       "      <td>European/Caucasian-American</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>[0-4]</td>\n",
       "      <td>[0-3]</td>\n",
       "      <td>[3-5]</td>\n",
       "      <td>7.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>[6-8]</td>\n",
       "      <td>[5-6]</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>female</td>\n",
       "      <td>21.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[0-1]</td>\n",
       "      <td>Asian/Pacific Islander/Asian-American</td>\n",
       "      <td>European/Caucasian-American</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>[0-4]</td>\n",
       "      <td>[0-3]</td>\n",
       "      <td>[3-5]</td>\n",
       "      <td>7.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>[6-8]</td>\n",
       "      <td>[5-6]</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>female</td>\n",
       "      <td>21.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[0-1]</td>\n",
       "      <td>Asian/Pacific Islander/Asian-American</td>\n",
       "      <td>Asian/Pacific Islander/Asian-American</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>[0-4]</td>\n",
       "      <td>[0-3]</td>\n",
       "      <td>[3-5]</td>\n",
       "      <td>7.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[6-8]</td>\n",
       "      <td>[0-4]</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>female</td>\n",
       "      <td>21.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>[2-3]</td>\n",
       "      <td>Asian/Pacific Islander/Asian-American</td>\n",
       "      <td>European/Caucasian-American</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>[0-4]</td>\n",
       "      <td>[0-3]</td>\n",
       "      <td>[3-5]</td>\n",
       "      <td>7.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>[6-8]</td>\n",
       "      <td>[5-6]</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>female</td>\n",
       "      <td>21.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>[2-3]</td>\n",
       "      <td>Asian/Pacific Islander/Asian-American</td>\n",
       "      <td>Latino/Hispanic American</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>[0-4]</td>\n",
       "      <td>[0-3]</td>\n",
       "      <td>[3-5]</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>[6-8]</td>\n",
       "      <td>[5-6]</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 120 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  has_null  wave  gender   age  age_o  d_age d_d_age  \\\n",
       "0        0   1.0  female  21.0   27.0    6.0   [4-6]   \n",
       "1        0   1.0  female  21.0   22.0    1.0   [0-1]   \n",
       "2        1   1.0  female  21.0   22.0    1.0   [0-1]   \n",
       "3        0   1.0  female  21.0   23.0    2.0   [2-3]   \n",
       "4        0   1.0  female  21.0   24.0    3.0   [2-3]   \n",
       "\n",
       "                                    race  \\\n",
       "0  Asian/Pacific Islander/Asian-American   \n",
       "1  Asian/Pacific Islander/Asian-American   \n",
       "2  Asian/Pacific Islander/Asian-American   \n",
       "3  Asian/Pacific Islander/Asian-American   \n",
       "4  Asian/Pacific Islander/Asian-American   \n",
       "\n",
       "                                  race_o samerace  ...  \\\n",
       "0            European/Caucasian-American        0  ...   \n",
       "1            European/Caucasian-American        0  ...   \n",
       "2  Asian/Pacific Islander/Asian-American        1  ...   \n",
       "3            European/Caucasian-American        0  ...   \n",
       "4               Latino/Hispanic American        0  ...   \n",
       "\n",
       "   expected_num_interested_in_me  expected_num_matches  \\\n",
       "0                            2.0                   4.0   \n",
       "1                            2.0                   4.0   \n",
       "2                            2.0                   4.0   \n",
       "3                            2.0                   4.0   \n",
       "4                            2.0                   4.0   \n",
       "\n",
       "  d_expected_happy_with_sd_people d_expected_num_interested_in_me  \\\n",
       "0                           [0-4]                           [0-3]   \n",
       "1                           [0-4]                           [0-3]   \n",
       "2                           [0-4]                           [0-3]   \n",
       "3                           [0-4]                           [0-3]   \n",
       "4                           [0-4]                           [0-3]   \n",
       "\n",
       "  d_expected_num_matches  like  guess_prob_liked  d_like  d_guess_prob_liked  \\\n",
       "0                  [3-5]   7.0               6.0   [6-8]               [5-6]   \n",
       "1                  [3-5]   7.0               5.0   [6-8]               [5-6]   \n",
       "2                  [3-5]   7.0               NaN   [6-8]               [0-4]   \n",
       "3                  [3-5]   7.0               6.0   [6-8]               [5-6]   \n",
       "4                  [3-5]   6.0               6.0   [6-8]               [5-6]   \n",
       "\n",
       "   met  \n",
       "0  0.0  \n",
       "1  1.0  \n",
       "2  1.0  \n",
       "3  0.0  \n",
       "4  0.0  \n",
       "\n",
       "[5 rows x 120 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nDAyrpysGXMg"
   },
   "source": [
    "A few things come up pretty quickly looking at this dataset. We have a lot of categorical features. So for modeling, we will need to encode them numerically. Some of these are actually encoded ranges. We will have to encode them specifically. More for exercise and illustration than for practical purpose, we are going to implement a transformer to plug into a sklearn pipeline in order to convert these range features to numeric features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "at2gsvibG8Bx",
    "outputId": "9641afa9-bb01-4ccd-a37d-c3f82683f8c0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['[0-0.33]', '[0.33-1]', '[-1-0]']\n",
       "Categories (3, object): ['[0-0.33]' < '[0.33-1]' < '[-1-0]']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_ranges = X['d_interests_correlate'].unique()\n",
    "example_ranges"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cQ7NWQ9px8v6"
   },
   "source": [
    "We can parse the ranges pretty easily in a function taking the lower and upper limit. And later on, we can then convert the ranges to something else, for example taking the mean of the range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "eGEXy3isGtDT"
   },
   "outputs": [],
   "source": [
    "def encode_ranges(range_str):\n",
    "  splits = range_str[1:-1].split('-')\n",
    "  range_max = splits[-1]\n",
    "  range_min = '-'.join(splits[:-1])\n",
    "  return range_min, range_max"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "p_0dF2qNyTlU"
   },
   "source": [
    "Let us try encoding these example ranges.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "-TZsR2UIyVQ0",
    "outputId": "de35466d-a2c6-4135-b56d-b4d570d50456"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('0', '0.33'), ('0.33', '1'), ('-1', '0')]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[encode_ranges(r) for r in example_ranges]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 952
    },
    "colab_type": "code",
    "id": "GC5Gi9goO6O9",
    "outputId": "e0a726f8-a0ef-4939-e270-a23eed8a5999"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['d_d_age',\n",
      " 'd_importance_same_race',\n",
      " 'd_importance_same_religion',\n",
      " 'd_pref_o_attractive',\n",
      " 'd_pref_o_sincere',\n",
      " 'd_pref_o_intelligence',\n",
      " 'd_pref_o_funny',\n",
      " 'd_pref_o_ambitious',\n",
      " 'd_pref_o_shared_interests',\n",
      " 'd_attractive_o',\n",
      " 'd_sinsere_o',\n",
      " 'd_intelligence_o',\n",
      " 'd_funny_o',\n",
      " 'd_ambitous_o',\n",
      " 'd_shared_interests_o',\n",
      " 'd_attractive_important',\n",
      " 'd_sincere_important',\n",
      " 'd_intellicence_important',\n",
      " 'd_funny_important',\n",
      " 'd_ambtition_important',\n",
      " 'd_shared_interests_important',\n",
      " 'd_attractive',\n",
      " 'd_sincere',\n",
      " 'd_intelligence',\n",
      " 'd_funny',\n",
      " 'd_ambition',\n",
      " 'd_attractive_partner',\n",
      " 'd_sincere_partner',\n",
      " 'd_intelligence_partner',\n",
      " 'd_funny_partner',\n",
      " 'd_ambition_partner',\n",
      " 'd_shared_interests_partner',\n",
      " 'd_sports',\n",
      " 'd_tvsports',\n",
      " 'd_exercise',\n",
      " 'd_dining',\n",
      " 'd_museums',\n",
      " 'd_art',\n",
      " 'd_hiking',\n",
      " 'd_gaming',\n",
      " 'd_clubbing',\n",
      " 'd_reading',\n",
      " 'd_tv',\n",
      " 'd_theater',\n",
      " 'd_movies',\n",
      " 'd_concerts',\n",
      " 'd_music',\n",
      " 'd_shopping',\n",
      " 'd_yoga',\n",
      " 'd_interests_correlate',\n",
      " 'd_expected_happy_with_sd_people',\n",
      " 'd_expected_num_interested_in_me',\n",
      " 'd_expected_num_matches',\n",
      " 'd_like',\n",
      " 'd_guess_prob_liked']\n"
     ]
    }
   ],
   "source": [
    "range_features = [\n",
    "                  col for col in X.select_dtypes(include='category')\n",
    "                  if X[col].apply(lambda x: x.startswith('[')\n",
    "                  if isinstance(x, str) else False).any()\n",
    "]\n",
    "\n",
    "import pprint\n",
    "pprint.pprint(range_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "I2-mVZ_cLcPn"
   },
   "source": [
    "But first, since we plan to make this preprocessing replicable in order to submit it to the competition, we will have to rewrite our preprocessing function as a scikit-learn transformer.\n",
    "\n",
    "Later on, we are going to submit this model to the OpenML competition dedicated to this dataset. OpenML is an organization that intends to make data science and machine learning more replicable and therefore more conducive to open research. Not only results are shown, but also the models are transparent. Therefore, during submission, you will have to use publicly available modules. This means that we had to create a module and publish it in the pypi python package repository. Here is the basic code for the RangeTransformer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9RkMOue3PUrH"
   },
   "source": [
    "```python\n",
    "from sklearn.base import BaseEstimator, TransformerMixin\n",
    "\n",
    "\n",
    "class RangeTransformer(BaseEstimator, TransformerMixin):\n",
    "    '''\n",
    "    A custom transformer for ranges.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    range_features : list[str]\n",
    "        This specifies the column names with the ranges. \n",
    "\n",
    "    Attributes\n",
    "    ----------\n",
    "    range_features : list[str]\n",
    "        Here we store the columns with range features.\n",
    "    suffix : this determines how we will rename the transformed features.\n",
    "    '''\n",
    "    def __init__(self, range_features=[], suffix='_range/mean'):\n",
    "        assert isinstance(range_features, list)\n",
    "        self.range_features = range_features\n",
    "        self.suffix = suffix\n",
    "    \n",
    "    def fit(self, X, y=None):\n",
    "        '''Nothing to do here\n",
    "        '''\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X, y=None):\n",
    "        range_data = pd.DataFrame()\n",
    "        for col in self.range_features:\n",
    "            range_data[str(col) + self.suffix] = X[col].apply(\n",
    "                lambda x: self._encode_ranges(x)\n",
    "            ).astype(float)\n",
    "        return range_data\n",
    "\n",
    "    @staticmethod\n",
    "    def _encode_ranges(range_str):\n",
    "        splits = range_str[1:-1].split('-')\n",
    "        range_max = float(splits[-1])\n",
    "        range_min = float('-'.join(splits[:-1]))\n",
    "        return sum([range_min, range_max]) / 2.0\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qYGP0JGFRrt6"
   },
   "source": [
    "The important thing about custom transformers in scikit-learn is that they have to inherit from *BaseEstimator* and *TransformerMixin* and that they have to implement *fit* and *transform* methods.\n",
    "\n",
    "Let us put this to the test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7gdFRC8MZS3o"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install -q openml_speed_dating_pipeline_steps==0.2.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nixxKZ_0Y_Um"
   },
   "outputs": [],
   "source": [
    "import openml_speed_dating_pipeline_steps as pipeline_steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Zlo9Ivx6Uvn6"
   },
   "outputs": [],
   "source": [
    "range_transformer = pipeline_steps.RangeTransformer(range_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JeAezMzZLMqO"
   },
   "source": [
    "Usually with a transformer, we would call the *fit()* or *fit_transform()* methods before any *transform()* calls, but since our *fit()* method does not do anything we can skip this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "X2pMe0dWU__3",
    "outputId": "60697c5f-06b7-48d0-8f29-0dc4a2ff3353"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 8378 entries, 0 to 8377\n",
      "Data columns (total 55 columns):\n",
      " #   Column                                      Non-Null Count  Dtype  \n",
      "---  ------                                      --------------  -----  \n",
      " 0   d_d_age_range/mean                          8378 non-null   float64\n",
      " 1   d_importance_same_race_range/mean           8378 non-null   float64\n",
      " 2   d_importance_same_religion_range/mean       8378 non-null   float64\n",
      " 3   d_pref_o_attractive_range/mean              8378 non-null   float64\n",
      " 4   d_pref_o_sincere_range/mean                 8378 non-null   float64\n",
      " 5   d_pref_o_intelligence_range/mean            8378 non-null   float64\n",
      " 6   d_pref_o_funny_range/mean                   8378 non-null   float64\n",
      " 7   d_pref_o_ambitious_range/mean               8378 non-null   float64\n",
      " 8   d_pref_o_shared_interests_range/mean        8378 non-null   float64\n",
      " 9   d_attractive_o_range/mean                   8378 non-null   float64\n",
      " 10  d_sinsere_o_range/mean                      8378 non-null   float64\n",
      " 11  d_intelligence_o_range/mean                 8378 non-null   float64\n",
      " 12  d_funny_o_range/mean                        8378 non-null   float64\n",
      " 13  d_ambitous_o_range/mean                     8378 non-null   float64\n",
      " 14  d_shared_interests_o_range/mean             8378 non-null   float64\n",
      " 15  d_attractive_important_range/mean           8378 non-null   float64\n",
      " 16  d_sincere_important_range/mean              8378 non-null   float64\n",
      " 17  d_intellicence_important_range/mean         8378 non-null   float64\n",
      " 18  d_funny_important_range/mean                8378 non-null   float64\n",
      " 19  d_ambtition_important_range/mean            8378 non-null   float64\n",
      " 20  d_shared_interests_important_range/mean     8378 non-null   float64\n",
      " 21  d_attractive_range/mean                     8378 non-null   float64\n",
      " 22  d_sincere_range/mean                        8378 non-null   float64\n",
      " 23  d_intelligence_range/mean                   8378 non-null   float64\n",
      " 24  d_funny_range/mean                          8378 non-null   float64\n",
      " 25  d_ambition_range/mean                       8378 non-null   float64\n",
      " 26  d_attractive_partner_range/mean             8378 non-null   float64\n",
      " 27  d_sincere_partner_range/mean                8378 non-null   float64\n",
      " 28  d_intelligence_partner_range/mean           8378 non-null   float64\n",
      " 29  d_funny_partner_range/mean                  8378 non-null   float64\n",
      " 30  d_ambition_partner_range/mean               8378 non-null   float64\n",
      " 31  d_shared_interests_partner_range/mean       8378 non-null   float64\n",
      " 32  d_sports_range/mean                         8378 non-null   float64\n",
      " 33  d_tvsports_range/mean                       8378 non-null   float64\n",
      " 34  d_exercise_range/mean                       8378 non-null   float64\n",
      " 35  d_dining_range/mean                         8378 non-null   float64\n",
      " 36  d_museums_range/mean                        8378 non-null   float64\n",
      " 37  d_art_range/mean                            8378 non-null   float64\n",
      " 38  d_hiking_range/mean                         8378 non-null   float64\n",
      " 39  d_gaming_range/mean                         8378 non-null   float64\n",
      " 40  d_clubbing_range/mean                       8378 non-null   float64\n",
      " 41  d_reading_range/mean                        8378 non-null   float64\n",
      " 42  d_tv_range/mean                             8378 non-null   float64\n",
      " 43  d_theater_range/mean                        8378 non-null   float64\n",
      " 44  d_movies_range/mean                         8378 non-null   float64\n",
      " 45  d_concerts_range/mean                       8378 non-null   float64\n",
      " 46  d_music_range/mean                          8378 non-null   float64\n",
      " 47  d_shopping_range/mean                       8378 non-null   float64\n",
      " 48  d_yoga_range/mean                           8378 non-null   float64\n",
      " 49  d_interests_correlate_range/mean            8378 non-null   float64\n",
      " 50  d_expected_happy_with_sd_people_range/mean  8378 non-null   float64\n",
      " 51  d_expected_num_interested_in_me_range/mean  8378 non-null   float64\n",
      " 52  d_expected_num_matches_range/mean           8378 non-null   float64\n",
      " 53  d_like_range/mean                           8378 non-null   float64\n",
      " 54  d_guess_prob_liked_range/mean               8378 non-null   float64\n",
      "dtypes: float64(55)\n",
      "memory usage: 3.5 MB\n"
     ]
    }
   ],
   "source": [
    "range_transformer.transform(X).info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7e2RARMqUxGX"
   },
   "source": [
    "Looks good. \n",
    "\n",
    "In the original version of the dataset - as presented in the book -, there was a lot more work to do: \n",
    "* missing values are represented as '?' in R format, and we would need another custom transformer would transform all features that contain only numbers or missing values ('?'). \n",
    "* This implies that in pandas/numpy numbers are encoded sometimes as strings. So we have to convert them. \n",
    "\n",
    "However, the version of the dataset on OpenML, already has missing values represented as *numpy.nan*, which lets us skip this conversion. You can see this preprocessor on github if you are interested.\n",
    "\n",
    "There is also another difference in the OpenML dataset to the original dataset: the original problem was easier because it included to additional features that represented the separate decisions by each of the two partners. This is absent here and therefore the task is more difficult. We saw performance drop drastically upon switching to the OpenML version of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "NcoMli06avp2",
    "outputId": "ad8f1d0b-b640-40e8-b52d-7c82805821a8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['pref_o_sincere',\n",
      " 'like',\n",
      " 'age_o',\n",
      " 'samerace',\n",
      " 'ambition',\n",
      " 'sincere',\n",
      " 'importance_same_race',\n",
      " 'music',\n",
      " 'exercise',\n",
      " 'clubbing',\n",
      " 'expected_happy_with_sd_people',\n",
      " 'age',\n",
      " 'theater',\n",
      " 'gaming',\n",
      " 'pref_o_shared_interests',\n",
      " 'field',\n",
      " 'race_o',\n",
      " 'sincere_important',\n",
      " 'attractive_o',\n",
      " 'ambtition_important',\n",
      " 'wave',\n",
      " 'tv',\n",
      " 'reading',\n",
      " 'tvsports',\n",
      " 'art',\n",
      " 'has_null',\n",
      " 'guess_prob_liked',\n",
      " 'attractive_partner',\n",
      " 'funny_important',\n",
      " 'shopping',\n",
      " 'expected_num_matches',\n",
      " 'hiking',\n",
      " 'yoga',\n",
      " 'attractive',\n",
      " 'movies',\n",
      " 'ambitous_o',\n",
      " 'interests_correlate',\n",
      " 'intelligence',\n",
      " 'sinsere_o',\n",
      " 'intelligence_partner',\n",
      " 'ambition_partner',\n",
      " 'sports',\n",
      " 'intelligence_o',\n",
      " 'importance_same_religion',\n",
      " 'shared_interests_important',\n",
      " 'shared_interests_partner',\n",
      " 'funny_partner',\n",
      " 'pref_o_ambitious',\n",
      " 'shared_interests_o',\n",
      " 'expected_num_interested_in_me',\n",
      " 'funny_o',\n",
      " 'dining',\n",
      " 'race',\n",
      " 'gender',\n",
      " 'd_age',\n",
      " 'pref_o_attractive',\n",
      " 'sincere_partner',\n",
      " 'pref_o_funny',\n",
      " 'funny',\n",
      " 'attractive_important',\n",
      " 'intellicence_important',\n",
      " 'pref_o_intelligence',\n",
      " 'museums',\n",
      " 'met',\n",
      " 'concerts']\n"
     ]
    }
   ],
   "source": [
    "remaining_columns = list(set(X.columns) - set(range_features))\n",
    "pprint.pprint(remaining_columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "V37fl55sDZF4"
   },
   "source": [
    "Let us look at the string features out of these remaining columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "oYa3V-_0eHFP",
    "outputId": "73609c5d-e90d-4ac7-d13e-e67dd2736db7"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>samerace</th>\n",
       "      <th>field</th>\n",
       "      <th>race_o</th>\n",
       "      <th>has_null</th>\n",
       "      <th>race</th>\n",
       "      <th>gender</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Law</td>\n",
       "      <td>European/Caucasian-American</td>\n",
       "      <td>0</td>\n",
       "      <td>Asian/Pacific Islander/Asian-American</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>Law</td>\n",
       "      <td>European/Caucasian-American</td>\n",
       "      <td>0</td>\n",
       "      <td>Asian/Pacific Islander/Asian-American</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Law</td>\n",
       "      <td>Asian/Pacific Islander/Asian-American</td>\n",
       "      <td>1</td>\n",
       "      <td>Asian/Pacific Islander/Asian-American</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>Law</td>\n",
       "      <td>European/Caucasian-American</td>\n",
       "      <td>0</td>\n",
       "      <td>Asian/Pacific Islander/Asian-American</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>Law</td>\n",
       "      <td>Latino/Hispanic American</td>\n",
       "      <td>0</td>\n",
       "      <td>Asian/Pacific Islander/Asian-American</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  samerace field                                 race_o has_null  \\\n",
       "0        0   Law            European/Caucasian-American        0   \n",
       "1        0   Law            European/Caucasian-American        0   \n",
       "2        1   Law  Asian/Pacific Islander/Asian-American        1   \n",
       "3        0   Law            European/Caucasian-American        0   \n",
       "4        0   Law               Latino/Hispanic American        0   \n",
       "\n",
       "                                    race  gender  \n",
       "0  Asian/Pacific Islander/Asian-American  female  \n",
       "1  Asian/Pacific Islander/Asian-American  female  \n",
       "2  Asian/Pacific Islander/Asian-American  female  \n",
       "3  Asian/Pacific Islander/Asian-American  female  \n",
       "4  Asian/Pacific Islander/Asian-American  female  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_columns = list(X[remaining_columns].select_dtypes(include='category').columns)\n",
    "X[cat_columns].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gSljERnjeRzY"
   },
   "source": [
    "Fair enough - there are no ranges left. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "woDOvdRJG7B8"
   },
   "source": [
    "We have already encountered the category encoder library. We will use it again here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "xqtMaacjD_S4",
    "outputId": "a60f7bd8-7cee-4384-8bc7-69e5a6c39a6f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install -q category_encoders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iXKP4xsHJKOV"
   },
   "source": [
    "Now we only have a few columns left. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 224
    },
    "colab_type": "code",
    "id": "Z50HOYAkJJYU",
    "outputId": "0ed566fa-6cb4-421e-b325-1a4e9dd5ffe2"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>like</th>\n",
       "      <th>age_o</th>\n",
       "      <th>ambition</th>\n",
       "      <th>sincere</th>\n",
       "      <th>music</th>\n",
       "      <th>exercise</th>\n",
       "      <th>age</th>\n",
       "      <th>sincere_important</th>\n",
       "      <th>pref_o_shared_interests</th>\n",
       "      <th>gaming</th>\n",
       "      <th>...</th>\n",
       "      <th>intelligence_o</th>\n",
       "      <th>funny_partner</th>\n",
       "      <th>pref_o_ambitious</th>\n",
       "      <th>shared_interests_o</th>\n",
       "      <th>expected_num_interested_in_me</th>\n",
       "      <th>dining</th>\n",
       "      <th>pref_o_attractive</th>\n",
       "      <th>funny</th>\n",
       "      <th>museums</th>\n",
       "      <th>met</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>8.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>10.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>10.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 59 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   like  age_o  ambition  sincere  music  exercise   age  sincere_important  \\\n",
       "0   7.0   27.0       7.0      8.0    9.0       8.0  21.0               20.0   \n",
       "1   7.0   22.0       7.0      8.0    9.0       8.0  21.0               20.0   \n",
       "2   7.0   22.0       7.0      8.0    9.0       8.0  21.0               20.0   \n",
       "3   7.0   23.0       7.0      8.0    9.0       8.0  21.0               20.0   \n",
       "4   6.0   24.0       7.0      8.0    9.0       8.0  21.0               20.0   \n",
       "\n",
       "   pref_o_shared_interests  gaming  ...  intelligence_o  funny_partner  \\\n",
       "0                      5.0     1.0  ...             8.0            7.0   \n",
       "1                      0.0     1.0  ...            10.0            8.0   \n",
       "2                     12.0     1.0  ...            10.0            8.0   \n",
       "3                      5.0     1.0  ...             9.0            7.0   \n",
       "4                     20.0     1.0  ...             9.0            7.0   \n",
       "\n",
       "   pref_o_ambitious  shared_interests_o  expected_num_interested_in_me  \\\n",
       "0               0.0                 6.0                            2.0   \n",
       "1               0.0                 5.0                            2.0   \n",
       "2              14.0                10.0                            2.0   \n",
       "3               5.0                 8.0                            2.0   \n",
       "4              10.0                 7.0                            2.0   \n",
       "\n",
       "   dining  pref_o_attractive  funny  museums  met  \n",
       "0     9.0               35.0    8.0      1.0  0.0  \n",
       "1     9.0               60.0    8.0      1.0  1.0  \n",
       "2     9.0               19.0    8.0      1.0  1.0  \n",
       "3     9.0               30.0    8.0      1.0  0.0  \n",
       "4     9.0               30.0    8.0      1.0  0.0  \n",
       "\n",
       "[5 rows x 59 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "other_features = list(set(remaining_columns) - set(cat_columns))\n",
    "X[other_features].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5Io8xpbNJurM"
   },
   "source": [
    "These are numeric or boolean features. So we will leave them as they are."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "b6GKKXOgLsb-"
   },
   "source": [
    "In order to put all the transformers together in a pipeline we have again different options. \n",
    "\n",
    "We can use FeatureUnion or ColumnTransformer. The ColumnTransformer is especially handy for pandas dataframes or numpy arrays since it allows specifying different operations for different subsets of the features. Another option is the FeatureUnion which allows concatenating the results from different transformations. Finally, pipelines and these feature unions can also be nested.\n",
    "\n",
    "We are going to use the ColumnTransformer here. For a different way to chain our operations together, please have a look at the PandasPicker in our repository."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_TQnyI8tNLSs"
   },
   "source": [
    "As we said, these features have to be imputed, i.e. replaced by other values. Common methods for dealing with missing values include:\n",
    "* dropping instances\n",
    "* dropping attributes\n",
    "* imputing the attribute average such as\n",
    "* mean, median or mode for all missing values\n",
    "* using regression to impute attribute missing values\n",
    "\n",
    "We will use a modeling strategy to impute. Random forest models will be fitted to the data to predict the values of missing values based on what is available. This will be done iteratively, i.e. first we will impute easy rows - ones that only miss a few values -, then we will impute ones that miss more values. Since this is all done under the hood all we need to use is the sklearn *IterativeImputer*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BqeeyXuqImVS"
   },
   "source": [
    "Another difficulty is that while the downloaded OpenML dataset can be viewed as a dataframe, when we upload the pipeline it is actually a numpy object. Therefore, we cannot use the column names, but we have to use numpy index slices to address the columns. Here is a utility function to convert between column names and numpy index slices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 197
    },
    "colab_type": "code",
    "id": "Fm1NzDmCIiV4",
    "outputId": "df14ee79-da18-42cf-d2a4-901548649faa"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([['0', 1.0, 'female', ..., '[6-8]', '[5-6]', 0.0],\n",
       "       ['0', 1.0, 'female', ..., '[6-8]', '[7-10]', 0.0],\n",
       "       ['0', 1.0, 'female', ..., '[6-8]', '[7-10]', 0.0],\n",
       "       ...,\n",
       "       ['0', 1.0, 'male', ..., '[6-8]', '[0-4]', 1.0],\n",
       "       ['0', 1.0, 'male', ..., '[9-10]', '[0-4]', 0.0],\n",
       "       ['0', 1.0, 'male', ..., '[6-8]', '[0-4]', 0.0]], dtype=object)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def col_names2slices(column_names, X):\n",
    "    numpy_slice = []\n",
    "    X_cols = list(X.columns)\n",
    "    for col in column_names:\n",
    "        numpy_slice.append(X_cols.index(col))\n",
    "    return tuple(numpy_slice)\n",
    "\n",
    "X.values[col_names2slices(list(range_features), X), :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QAj_oao-QQj3"
   },
   "outputs": [],
   "source": [
    "from sklearn.experimental import enable_iterative_imputer  \n",
    "from sklearn.impute import IterativeImputer\n",
    "from sklearn.compose import ColumnTransformer\n",
    "import category_encoders as ce\n",
    "\n",
    "\n",
    "ordinal_encoder = ce.OrdinalEncoder(\n",
    "    cols=None,  # all features that it encounters\n",
    "    handle_missing='return_nan',\n",
    "    handle_unknown='ignore'\n",
    ").fit(X)\n",
    "\n",
    "imputer = IterativeImputer(\n",
    "    imputation_order='ascending',  # also try 'descending'\n",
    "    random_state=0,\n",
    "    max_iter=10  # try higher values\n",
    ")\n",
    "\n",
    "range_transformer = pipeline_steps.RangeTransformer(\n",
    "    range_features\n",
    ")\n",
    "\n",
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        #('ranges', range_transformer, range_features),\n",
    "        ('cat', ordinal_encoder, categorical_features)],\n",
    "    remainder='passthrough'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "13seev66e8RY"
   },
   "source": [
    "This is our target column, which represents whether the pairing was successful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "nz7Bi-xie2CE",
    "outputId": "5e99c574-e8ed-45e5-d9bb-0c5cb0b34d11"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['0', '1']\n",
       "Categories (2, object): ['0' < '1']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 296
    },
    "colab_type": "code",
    "id": "wTuPI3qOQxIE",
    "outputId": "44f876b0-920f-4b4c-d206-87c338239b73"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Text(0, 0.5, 'Relative frequency'), Text(0.5, 0, 'Match')]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAVnUlEQVR4nO3de5QmdX3n8feHiwGUm9ISDpcMKkII4Eha1MMaV5SEoIImrEIwYsJhdhPco9GTVUy8x93oqiRusomDoCMRg2IME4JJEEY8ZhUcZBxuuhJQd5BIm4AgJiD43T+qRpqhLzXdXc9DT71f5zyn6/LUU9+ih0/X86uq3y9VhSRpOLYbdwGSpNEy+CVpYAx+SRoYg1+SBsbgl6SB2WHcBXSx11571YoVK8ZdhiQtK9dcc833qmpiy+XLIvhXrFjB+vXrx12GJC0rSb4103KbeiRpYAx+SRoYg1+SBsbgl6SBMfglaWAMfkkaGINfkgbG4JekgTH4JWlglsWTu4txwVXfnnH5rz3zgBFXIkmPDp7xS9LAGPySNDAGvyQNjMEvSQPTe/An2T7JtUkuaecPTHJVkpuTXJjkMX3XIEl6yCjO+F8D3DRt/t3A2VX1FOBO4PQR1CBJavUa/En2A14IfKidD3AMcFH7ljXAS/qsQZL0cH2f8f8R8N+AH7fzTwDuqqoH2vlNwL4zbZhkVZL1SdZPTU31XKYkDUdvwZ/kRcAdVXXNQravqtVVNVlVkxMTjxgyUpK0QH0+uXs0cEKS44GdgN2APwb2SLJDe9a/H3BbjzVIkrbQ2xl/VZ1VVftV1QrgZOCKqjoVWAec1L7tNODivmqQJD3SOO7jfwPwuiQ307T5nzuGGiRpsEbSSVtVfQ74XDt9C3DUKPYrSXokn9yVpIEx+CVpYAx+SRoYg1+SBsbgl6SBMfglaWAMfkkaGINfkgbG4JekgTH4JWlgDH5JGhiDX5IGxuCXpIEx+CVpYAx+SRoYg1+SBqbPwdZ3SnJ1kq8muSHJ29vlH0lya5IN7WtlXzVIkh6pzxG47gOOqaofJNkR+EKSz7TrfreqLupx35KkWfQW/FVVwA/a2R3bV/W1P0lSN7228SfZPskG4A7gsqq6ql31riQbk5yd5Kdm2XZVkvVJ1k9NTfVZpiQNSq/BX1UPVtVKYD/gqCSHAWcBhwDPAB4PvGGWbVdX1WRVTU5MTPRZpiQNykju6qmqu4B1wHFVdXs17gM+DBw1ihokSY0+7+qZSLJHO70zcCzwtST7tMsCvAS4vq8aJEmP1OddPfsAa5JsT/MH5hNVdUmSK5JMAAE2AP+lxxokSVvo866ejcDTZ1h+TF/7lCTNzyd3JWlgDH5JGhiDX5IGxuCXpIEx+CVpYAx+SRoYg1+SBsbgl6SBMfglaWAMfkkaGINfkgbG4JekgTH4JWlgDH5JGph5gz/J4aMoRJI0Gl3O+P93kquT/HaS3XuvSJLUq3mDv6qeA5wK7A9ck+SCJMfOt12Sndo/GF9NckOSt7fLD0xyVZKbk1yY5DGLPgpJUmed2vir6hvA7wNvAJ4LfCDJ15L8yhyb3QccU1VPA1YCxyV5FvBu4OyqegpwJ3D6Yg5AkrR1urTxH5HkbOAm4BjgxVX1s+302bNtV40ftLM7tq9qt7uoXb6GZsB1SdKIdDnj/1/AV4CnVdWZVfUVgKr6Ds23gFkl2T7JBuAO4DLgn4C7quqB9i2bgH1n2XZVkvVJ1k9NTXU7GknSvLoE/wuBC6rq3wCSbJdkF4CqOn+uDavqwapaCewHHAUc0rWwqlpdVZNVNTkxMdF1M0nSPLoE/2eBnafN79Iu66yq7gLWAc8G9kiyQ7tqP+C2rfksSdLidAn+naa11dNO7zLfRkkmkuzRTu8MHEtznWAdcFL7ttOAi7e2aEnSwnUJ/nuTHLl5JsnPA//WYbt9gHVJNgJfBi6rqkto7gx6XZKbgScA52592ZKkhdph/rfwWuCTSb4DBPhp4OXzbVRVG4Gnz7D8Fpr2fknSGMwb/FX15SSHAAe3i75eVT/qtyxJUl+6nPEDPANY0b7/yCRU1Ud7q0qS1Jt5gz/J+cCTgQ3Ag+3iAgx+SVqGupzxTwKHVlX1XYwkqX9d7uq5nuaCriRpG9DljH8v4MYkV9N0vAZAVZ3QW1WSpN50Cf639V2EJGl0utzOeWWSnwEOqqrPtv30bN9/aZKkPnTplvkMmm6UP9gu2hf46z6LkiT1p8vF3TOBo4G74SeDsjyxz6IkSf3pEvz3VdX9m2fanjW9tVOSlqkuwX9lkjcBO7dj7X4S+Jt+y5Ik9aVL8L8RmAKuA/4zcCnzjLwlSXr06nJXz4+Bc9qXJGmZ69JXz63M0KZfVU/qpSJJUq+69tWz2U7AfwIe3085kqS+zdvGX1X/Mu11W1X9Ec0A7HNKsn+SdUluTHJDkte0y9+W5LYkG9rX8UtwHJKkjro09Rw5bXY7mm8AXb4pPAC8vqq+kmRX4Jokl7Xrzq6q9251tZKkResS4O+bNv0A8E3gZfNtVFW3A7e30/ckuYnmqV9J0hh1uavneYvdSZIVNOPvXkXzFPCrk7wSWE/zreDOGbZZBawCOOCAAxZbgiSp1aWp53Vzra+q98+z/eOATwGvraq7k/wZ8E6aO4XeSfON4jdn+NzVwGqAyclJnxSWpCXS9a6eZwBr2/kXA1cD35hvwyQ70oT+x6rqrwCq6rvT1p8DXLKVNUuSFqFL8O8HHFlV90BzVw7wt1X1irk2ShLgXOCm6d8KkuzTtv8DvJRmhC9J0oh0Cf69gfunzd/fLpvP0cCvA9cl2dAuexNwSpKVNE0936TpBkKSNCJdgv+jwNVJPt3OvwRYM99GVfUFIDOsurR7eZKkpdblrp53JfkM8Jx20W9U1bX9liVJ6kuX3jkBdgHurqo/BjYlObDHmiRJPeoy9OJbgTcAZ7WLdgT+os+iJEn96XLG/1LgBOBegKr6DrBrn0VJkvrTJfjvr6qi7Zo5yWP7LUmS1Kcuwf+JJB8E9khyBvBZHJRFkpatOe/qaR/CuhA4BLgbOBh4S1VdNtd2kqRHrzmDv6oqyaVVdThg2EvSNqBLU89Xkjyj90okSSPR5cndZwKnJvkWzZ09ofkycESvlUmSejFr8Cc5sKpuBX5phPVIkno21xn/RcDPA+dV1fNHVI8kqWdzBf92Sd4EPHWmwVjmG4BFkvToNNfF3ZOBB2n+OOw6w0uStAzNesZfVV8H3p1kY1V9ZoQ1SZJ6NO/tnAsN/ST7J1mX5MYkNyR5Tbv88UkuS/KN9ueeC/l8SdLCdO2WeSEeAF5fVYcCzwLOTHIo8Ebg8qo6CLi8nZckjUhvwV9Vt1fVV9rpe4CbgH2BE3loBK81NCN6SZJGpEt//LskeXOSc9r5g5K8aGt2kmQF8HTgKmDvaYOt/zPdxu+VJC2RLmf8HwbuA57dzt8G/EHXHSR5HPAp4LVVdff0ddO7e55hu1VJ1idZPzU11XV3kqR5dAn+J1fVe4AfAVTVD5l5EPVHSLIjTeh/rKr+ql383ST7tOv3Ae6YaduqWl1Vk1U1OTEx0WV3kqQOOg3EkmRnHhqI5ck03wDm1HbpfC5w0xYPe60FTmunTwMu3qqKJUmL0qWTtrcBfwfsn+RjwNHAqzpsdzTw68B1STa0y94E/CHN4C6nA98CXraVNUuSFmHe4K+qf0hyDc0tmQFeU1Xf67DdF5i9Sci+fyRpTOYN/iR/A1wArK2qe/svSZLUpy5t/O8FngPcmOSiJCcl2annuiRJPenS1HMlcGWS7YFjgDOA84Ddeq5NktSDLhd3ae/qeTHwcuBIHnryVpK0zHRp4/8EcBTNnT1/AlxZVT/uuzBJUj+6nPGfC5xSVQ/2XYwkqX9zjbl7TFVdATwWOLF5Hush057ElSQtI3Od8T8XuIKmbX9LBRj8krQMzTUC11vbyXdU1a3T1yU5sNeqJEm96XIf/6dmWHbRUhciSRqNudr4DwF+Dtg9ya9MW7Ub4ANckrRMzdXGfzDwImAPHt7Ofw/NQ1ySpGVorjb+i4GLkzy7qr44wpokST3qch//tUnOpGn2+UkTT1X9Zm9VSZJ60+Xi7vnATwO/BFwJ7EfT3CNJWoa6BP9TqurNwL1VtQZ4IfDMfsuSJPWlS/D/qP15V5LDgN2BJ863UZLzktyR5Pppy96W5LYkG9rX8QsrW5K0UF2Cf3WSPYE304yXeyPwng7bfQQ4boblZ1fVyvZ1aedKJUlLokt//B9qJ68EntT1g6vq80lWLKwsSVJf5nqA63VzbVhV71/gPl+d5JXAeuD1VXXnLPtfBawCOOCAAxa4K0nSluZq6tl1ntdC/BnwZGAlcDvwvtneWFWrq2qyqiYnJiYWuDtJ0pbmeoDr7Uu9s6r67ubpJOcAlyz1PiRJc5v34m6Spya5fPPdOUmOSPL7C9lZkn2mzb4UuH6290qS+tHlrp5zgLNob+usqo3AyfNtlOTjwBeBg5NsSnI68J4k1yXZCDwP+J0FVy5JWpAuXTbsUlVXbzEC1wPzbVRVp8yw+NyuhUmS+tHljP97SZ5MM+oWSU6iuTArSVqGupzxnwmsBg5JchtwK3Bqr1VJknrT5QGuW4AXJHkszTeEH9K08X+r59okST2YtaknyW5JzkryJ0mOpQn804CbgZeNqkBJ0tKa64z/fOBOmjtzzgB+Dwjw0qraMILaJEk9mCv4n1RVhwMk+RDNBd0DqurfR1KZJKkXc93Vs7k7ZqrqQWCToS9Jy99cZ/xPS3J3Ox1g53Y+QFXVbr1XJ0lacnP11bP9KAuRJI1Glwe4JEnbEINfkgbG4JekgTH4JWlgDH5JGhiDX5IGxuCXpIHpLfiTnJfkjs1DNrbLHp/ksiTfaH/u2df+JUkz6/OM/yPAcVsseyNweVUdBFzezkuSRqi34K+qzwP/usXiE4E17fQa4CV97V+SNLNRt/HvXVWbh238Z2Dv2d6YZFWS9UnWT01NjaY6SRqAsV3craqiHcd3lvWrq2qyqiYnJiZGWJkkbdtGHfzfTbIPQPvzjhHvX5IGb9TBv5Zm+EbanxePeP+SNHh93s75cZphGw9OsinJ6cAfAscm+QbwgnZekjRCcw3EsihVdcosq57f1z4lSfPzyV1JGhiDX5IGxuCXpIHprY1fktTNBVd9e9Z1v/bMA5Z8f57xS9LAGPySNDAGvyQNjMEvSQNj8EvSwBj8kjQwBr8kDYzBL0kDY/BL0sAY/JI0MAa/JA2MwS9JAzOWTtqSfBO4B3gQeKCqJsdRhyQN0Th753xeVX1vjPuXpEGyqUeSBmZcwV/APyS5Jsmqmd6QZFWS9UnWT01Njbg8Sdp2jSv4/0NVHQn8MnBmkl/Y8g1VtbqqJqtqcmJiYvQVStI2aizBX1W3tT/vAD4NHDWOOiRpiEYe/Ekem2TXzdPALwLXj7oOSRqqcdzVszfw6SSb939BVf3dGOqQpEEaefBX1S3A00a9X0lSw9s5JWlgDH5JGhiDX5IGxuCXpIEx+CVpYAx+SRoYg1+SBsbgl6SBMfglaWAMfkkaGINfkgbG4JekgTH4JWlgDH5JGhiDX5IGxuCXpIEZS/AnOS7J15PcnOSN46hBkoZqHGPubg/8KfDLwKHAKUkOHXUdkjRU4zjjPwq4uapuqar7gb8EThxDHZI0SOMYbH1f4P9Nm98EPHPLNyVZBaxqZ3+Q5OsL3N9ewPe2XHjqAj9smZjxmLdxHvMwDO6YT13cMf/MTAvHEfydVNVqYPViPyfJ+qqaXIKSlg2PeRg85mHo45jH0dRzG7D/tPn92mWSpBEYR/B/GTgoyYFJHgOcDKwdQx2SNEgjb+qpqgeSvBr4e2B74LyquqHHXS66uWgZ8piHwWMehiU/5lTVUn+mJOlRzCd3JWlgDH5JGphtJvjn6wYiyU8lubBdf1WSFaOvcml1OObXJbkxycYklyeZ8Z7e5aRrdx9JfjVJJVnWt/51Od4kL2t/zzckuWDUNS61Dv+uD0iyLsm17b/t48dR51JKcl6SO5JcP8v6JPlA+99kY5IjF7XDqlr2L5qLxP8EPAl4DPBV4NAt3vPbwJ+30ycDF4677hEc8/OAXdrp3xrCMbfv2xX4PPAlYHLcdff8Oz4IuBbYs51/4rjrHsExrwZ+q50+FPjmuOteguP+BeBI4PpZ1h8PfAYI8CzgqsXsb1s54+/SDcSJwJp2+iLg+UkywhqX2rzHXFXrquqH7eyXaJ6ZWM66dvfxTuDdwL+PsrgedDneM4A/rao7AarqjhHXuNS6HHMBu7XTuwPfGWF9vaiqzwP/OsdbTgQ+Wo0vAXsk2Weh+9tWgn+mbiD2ne09VfUA8H3gCSOprh9djnm602nOGJazeY+5/Qq8f1X97SgL60mX3/FTgacm+cckX0py3Miq60eXY34b8Iokm4BLgf86mtLGamv/f5/To7bLBi2dJK8AJoHnjruWPiXZDng/8KoxlzJKO9A09/xHmm90n09yeFXdNdaq+nUK8JGqel+SZwPnJzmsqn487sKWi23ljL9LNxA/eU+SHWi+Iv7LSKrrR6euL5K8APg94ISqum9EtfVlvmPeFTgM+FySb9K0ha5dxhd4u/yONwFrq+pHVXUr8H9p/hAsV12O+XTgEwBV9UVgJ5qOzLZlS9rVzbYS/F26gVgLnNZOnwRcUe1Vk2Vq3mNO8nTggzShv9zbfmGeY66q71fVXlW1oqpW0FzXOKGq1o+n3EXr8u/6r2nO9kmyF03Tzy2jLHKJdTnmbwPPB0jyszTBPzXSKkdvLfDK9u6eZwHfr6rbF/ph20RTT83SDUSSdwDrq2otcC7NV8KbaS6inDy+ihev4zH/T+BxwCfb69jfrqoTxlb0InU85m1Gx+P9e+AXk9wIPAj8blUt22+yHY/59cA5SX6H5kLvq5b5SRxJPk7zB3yv9trFW4EdAarqz2muZRwP3Az8EPiNRe1vmf/3kiRtpW2lqUeS1JHBL0kDY/BL0sAY/JI0MAa/JA2MwS8BbU+efzFtfockU0kumWe7lV16h0zyg6WoU1oKBr/UuBc4LMnO7fyxdHsyciXN/dXSsmHwSw+5FHhhO30K8PHNK5IcleSLbR/w/yfJwe2Tpe8AXp5kQ5KXJ3lckg8nua7tN/1Xp33Gu5J8te1Mbe+RHpk0jcEvPeQvgZOT7AQcAVw1bd3XgOdU1dOBtwD/ve02+C004xysrKoLgTfTPE5/eFUdAVzRbv9Y4EtV9TSasQLOGM0hSY+0TXTZIC2FqtrYjsx2Cs3Z/3S7A2uSHETTTcCOs3zMC5jWHcjmfvKB+4HN1wuuoWlKksbCM37p4dYC72VaM0/rncC6qjoMeDFNx2Bb40fT+pN5EE+6NEYGv/Rw5wFvr6rrtli+Ow9d7H3VtOX30HQHvdllwJmbZ5Ls2UON0qIY/NI0VbWpqj4ww6r3AP8jybU8/Gx9HXDo5ou7wB8Aeya5PslXacY9lh5V7J1TkgbGM35JGhiDX5IGxuCXpIEx+CVpYAx+SRoYg1+SBsbgl6SB+f+lJwz0G9TnxQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "import pandas as pd\n",
    "%matplotlib inline\n",
    "import seaborn as sns\n",
    "\n",
    "ax = sns.distplot(y.astype(int), kde=False, norm_hist=True)\n",
    "ax.set(xlabel='Match', ylabel='Relative frequency')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wAIhGT4uQwnW"
   },
   "source": [
    "The humans in our dataset are a choosy bunch. A lot more non-matches than matches. It is probably worth resampling to get a more balanced distribution of the two classes. We finish off our pipeline by choosing the Balanced Random Forest classifier from the imbalanced learn package.\n",
    "\n",
    "Alternatively, we could have used the AdaSyn resampling strategy and add it to our pipeline. We have to use the pipeline implementation by imblearn, because the Sampling is incompatible with sklearn transformers - the imbalanced learn module uses different methods in the transformers such as *fit_resample()*. Talking about different pipeline implementations, the dask package has its own pipeline as well, which is compatible with dask objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cyJyWqnoclf1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already up-to-date: scikit-learn>=0.23 in /Users/ben/anaconda3/lib/python3.6/site-packages (0.23.2)\n",
      "Requirement already up-to-date: imbalanced-learn==0.7.0 in /Users/ben/anaconda3/lib/python3.6/site-packages (0.7.0)\n",
      "Requirement already satisfied, skipping upgrade: joblib>=0.11 in /Users/ben/anaconda3/lib/python3.6/site-packages (from scikit-learn>=0.23) (0.16.0)\n",
      "Requirement already satisfied, skipping upgrade: scipy>=0.19.1 in /Users/ben/anaconda3/lib/python3.6/site-packages (from scikit-learn>=0.23) (1.5.2)\n",
      "Requirement already satisfied, skipping upgrade: threadpoolctl>=2.0.0 in /Users/ben/anaconda3/lib/python3.6/site-packages (from scikit-learn>=0.23) (2.1.0)\n",
      "Requirement already satisfied, skipping upgrade: numpy>=1.13.3 in /Users/ben/anaconda3/lib/python3.6/site-packages (from scikit-learn>=0.23) (1.18.5)\n"
     ]
    }
   ],
   "source": [
    "from imblearn.ensemble import (\n",
    "    BalancedRandomForestClassifier,\n",
    "    RUSBoostClassifier,\n",
    "    BalancedBaggingClassifier\n",
    ")\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "def create_model(n_estimators=20):\n",
    "    return Pipeline(\n",
    "        steps=[('preprocessor', preprocessor),\n",
    "              ('imputation', imputer),\n",
    "              ('rf', BalancedBaggingClassifier(\n",
    "                  n_estimators=n_estimators,\n",
    "                  base_estimator=RandomForestClassifier()\n",
    "                  ))]\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yoLjx2ZGOkvv"
   },
   "source": [
    "Now we split training and test, and try it out!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "32iextMRUz4M"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "y = y.astype(int)\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "   X , y, test_size=0.33, random_state=42, stratify=y\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "b0uKLK2CuyCR",
    "outputId": "c8e403ad-328f-4f69-e9d7-2242cdb4534d"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ben/anaconda3/lib/python3.6/site-packages/category_encoders/utils.py:21: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead\n",
      "  elif pd.api.types.is_categorical(cols):\n",
      "/Users/ben/anaconda3/lib/python3.6/site-packages/sklearn/impute/_iterative.py:670: ConvergenceWarning: [IterativeImputer] Early stopping criterion not reached.\n",
      "  \" reached.\", ConvergenceWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "auc: 0.7884615384615384\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ben/anaconda3/lib/python3.6/site-packages/category_encoders/utils.py:21: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead\n",
      "  elif pd.api.types.is_categorical(cols):\n",
      "/Users/ben/anaconda3/lib/python3.6/site-packages/sklearn/impute/_iterative.py:670: ConvergenceWarning: [IterativeImputer] Early stopping criterion not reached.\n",
      "  \" reached.\", ConvergenceWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "auc: 0.7858641358641358\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ben/anaconda3/lib/python3.6/site-packages/category_encoders/utils.py:21: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead\n",
      "  elif pd.api.types.is_categorical(cols):\n",
      "/Users/ben/anaconda3/lib/python3.6/site-packages/sklearn/impute/_iterative.py:670: ConvergenceWarning: [IterativeImputer] Early stopping criterion not reached.\n",
      "  \" reached.\", ConvergenceWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "auc: 0.7856143856143856\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ben/anaconda3/lib/python3.6/site-packages/category_encoders/utils.py:21: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead\n",
      "  elif pd.api.types.is_categorical(cols):\n",
      "/Users/ben/anaconda3/lib/python3.6/site-packages/sklearn/impute/_iterative.py:670: ConvergenceWarning: [IterativeImputer] Early stopping criterion not reached.\n",
      "  \" reached.\", ConvergenceWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "auc: 0.7836663336663336\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ben/anaconda3/lib/python3.6/site-packages/category_encoders/utils.py:21: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead\n",
      "  elif pd.api.types.is_categorical(cols):\n",
      "/Users/ben/anaconda3/lib/python3.6/site-packages/sklearn/impute/_iterative.py:670: ConvergenceWarning: [IterativeImputer] Early stopping criterion not reached.\n",
      "  \" reached.\", ConvergenceWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "auc: 0.7869463869463869\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score, confusion_matrix\n",
    "\n",
    "performances = []\n",
    "for n_estimators in [20, 50, 100, 200, 500]:\n",
    "    resampling_pipeline_bclf = create_model(n_estimators)\n",
    "    resampling_pipeline_bclf.fit(X_train, y_train)\n",
    "    y_predicted = resampling_pipeline_bclf.predict(X_test)\n",
    "    auc = roc_auc_score(y_test, y_predicted)\n",
    "    print('auc: {}'.format(auc))\n",
    "    performances.append(auc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "-J29UfTvZI4C",
    "outputId": "1b89abc6-ffa9-4189-bf1f-3cd7365a0536"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.7884615384615384, 0.7858641358641358, 0.7856143856143856, 0.7836663336663336, 0.7869463869463869]\n"
     ]
    }
   ],
   "source": [
    "print(performances)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MVsJWI5yVh6M"
   },
   "outputs": [],
   "source": [
    "y_predicted = resampling_pipeline_bclf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 129
    },
    "colab_type": "code",
    "id": "TYh0dwbfingK",
    "outputId": "88861402-337f-4ab5-db9b-b9a54d86d959"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "auc: 0.7869463869463869\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[1874,  436],\n",
       "       [ 108,  347]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score, confusion_matrix\n",
    "print('auc: {}'.format(roc_auc_score(y_test, y_predicted)))\n",
    "confusion_matrix(y_test, y_predicted, labels=[0, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7n5NVhTchLW8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1874,  436],\n",
       "       [ 108,  347]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix(y_test, y_predicted, labels=[0, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MQt6lwetbKAI"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Predicting_partner_choices.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
